%%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': 'arial', 'fontSize': '10px'}}}%%
gantt
title Project Work Plan Gantt Chart
dateFormat YYYY-MM-DD
axisFormat %b '%y
todayMarker off
section Work Packages
WP1 Use case definition, data collection and process mapping :2024-11-01, 2024-12-31
WP2 Q/A extraction of unstructured regulatory text :2024-12-01, 2025-01-31
WP3 Automated rule extraction based on OWL ontologies :2025-02-01, 2025-05-31
WP4 Computational approaches to finding and measuring rule set inconsistencies :2025-06-01, 2025-08-31
WP5 Risk assessment for regulatory compliance :2025-09-01, 2025-10-31
section Economic Analysis Milestones
M1 Baseline Process Mapping & Initial CBA Report :milestone, 2024-12-31, 1d
M2 Q/A CBA Assessment :milestone, 2025-01-31, 1d
M3 Simulation Model for Scalability :milestone, 2025-05-31, 1d
M4 Qualitative Interviews & Final CBA :milestone, 2025-10-31, 1d
section Extended Timeline with Partner
Refinement & Integration Phase :2025-11-01, 2026-08-27
Functioning AI-enhanced module :milestone, 2026-02-28, 1d
Final AI-enhanced module :milestone, 2026-08-27, 1d
User Testing & Trials :2026-03-01, 2026-06-30
User Agreements :milestone, 2026-02-27, 1d
Trials Feedback Report :milestone, 2026-06-27, 1d
Bridging AI and Economics
Evaluating the Feasibility of RAG Models for Regulatory Compliance
Project Summary
This project, led by Dr Barry Quinn and Dr Jesus Martinez Del Rincon from Queen’s University Belfast, aims to evaluate the potential of AI in enhancing regulatory compliance within the investment management sector. Partnering with Funds Axis Ltd., this feasibility study will examine the use of Retrieval-Augmented Generation (RAG) models to support efficient, accurate interpretation and implementation of complex regulatory requirements.
Key Objectives
- Exploring AI Applications in Compliance: Assessing RAG models’ capacity to support compliance processes through improved information retrieval, accuracy, and interpretability.
- Economic and Operational Evaluation: Using a structured economic analysis to understand potential cost efficiencies, process enhancements, and operational feasibility for a FinTech firm.
- Focusing on Key Compliance Areas: Emphasising regulatory reporting, risk assessments, and compliance monitoring as areas where AI tools may add value.
- Addressing Core AI and Compliance Challenges: Identifying any key limitations, such as adaptability to changing regulations, interpretability, and operational integration, with a view to developing solutions where possible.
Analysis Approach
To assess the viability and potential benefits of RAG models, a hybrid economic analysis approach will be applied, combining:
• Cost-Benefit Analysis for a structured evaluation of cost impacts. • Process Mapping and Time-Motion Analysis to capture time efficiencies and workflow improvements. - Simulation Modelling to test model adaptability to evolving regulatory needs. • Qualitative Insights from key stakeholders to explore usability and functionality within compliance roles.
This analysis will aim to provide an evidence-based evaluation of the model’s feasibility for Funds Axis Ltd., focusing on practical applications and potential efficiencies within the existing compliance framework.
Funding and Duration
Supported by UK Research and Innovation (UKRI) through the UKFin+ programme, this 12-month project is set to run from 1 November 2024 to 31 October 2025. Through this study, the project team will generate insights that contribute to both practical implementation options and an understanding of the broader potential of RAG models in regulatory compliance.
Anticipated Outcomes
The expected outcomes include a clearer understanding of how RAG models might improve efficiency and support compliance goals within a FinTech setting. Findings from this study will aim to offer actionable insights into RAG’s practical, economic, and operational impact, providing a foundation for future AI integration within Funds Axis Ltd.
Hybrid Economic Analysis Approach
To assess the feasibility and potential benefits of RAG models within a FinTech setting, this project employs a hybrid economic analysis approach. This approach combines quantitative and qualitative methods to provide a balanced view of the operational and economic impacts of AI-driven compliance support. The key components include:
- Cost-Benefit Analysis (CBA) • Purpose: To estimate cost savings and efficiency gains associated with the RAG model. • Method: The CBA will examine both direct and indirect compliance costs, assessing time and resource expenditure in current processes compared with those supported by AI. • Anticipated Insights: Expected findings include projected cost reductions and resource efficiencies, which will help quantify RAG’s economic viability.
- Process Mapping and Time-Motion Analysis • Purpose: To identify workflow efficiencies and areas where RAG could improve task completion times. • Method: Documenting existing compliance workflows will help establish a baseline of time and resource requirements, enabling direct comparisons once RAG support is applied. • Anticipated Insights: This analysis will offer detailed insights into specific compliance tasks that may benefit most from AI support, helping to prioritise areas for implementation.
- Simulation Modelling • Purpose: To assess how well RAG models might adapt to changing regulatory requirements and differing compliance scenarios. • Method: Simulations based on typical compliance scenarios will allow exploration of how the RAG model performs under various conditions, including regulatory updates. • Anticipated Insights: By testing the model’s adaptability, this component will help determine whether RAG offers scalability potential in a broader compliance context.
- Qualitative Interviews • Purpose: To gain insight into RAG’s usability and value from a user perspective, gathering practical feedback from compliance professionals. • Method: Interviews with key compliance staff will capture experiences with the RAG model, focusing on ease of use, perceived efficiency, and operational impact. • Anticipated Insights: Qualitative feedback will add context to quantitative findings, highlighting any practical considerations, challenges, or adjustments needed to support effective use.
Summary of Approach
This hybrid analysis framework is designed to provide a comprehensive view of the RAG model’s practical and economic feasibility within compliance processes. By combining these methods, the study aims to provide Funds Axis Ltd. with actionable insights that reflect both cost and operational impacts, supporting evidence-based decision-making around AI adoption.
Work Packages
The project is divided into five work packages (WPs), each designed to address specific aspects of implementing and evaluating a RAG model in regulatory compliance. Each WP includes an economic analysis component to provide insights into the model’s impact on compliance costs and operational efficiency.
WP1: Definition of Use Cases, Data Collection, and Process Mapping
Timeline: Months 1–2
WP2: Question/Answer Extraction of Unstructured Regulatory Text
Timeline: Months 2–3
WP3: Rule Extraction Using OWL Ontology and Knowledge Base Integration
Timeline: Months 4–7
WP4: Computational Approaches to Identifying and Quantifying Inconsistencies in Rule Sets
Timeline: Months 8–10
WP5: Risk Assessment and Qualitative Insights from Compliance Officers
Timeline: Months 11–12
Each work package builds on the previous phase, creating a structured, cumulative assessment of the RAG model’s economic and operational viability. This approach provides Funds Axis Ltd. with practical insights into specific compliance improvements that may be achieved through AI support.
Theoretical Framework in Economics
This study uses a structured economic framework to evaluate the potential benefits and challenges of implementing RAG models in regulatory compliance. This framework is grounded in key economic principles relevant to the efficiency, scalability, and adaptability of AI solutions, especially within the resource-constrained setting of a small financial technology firm.
Core Economic Principles
- Efficiency and Cost-Effectiveness • Principle: Efficiency is central to evaluating the economic value of AI in compliance. The model’s ability to save time and reduce resource requirements directly impacts its cost-effectiveness. • Application: Cost-Benefit Analysis (CBA) and Process Mapping will quantify any potential savings from implementing the RAG model. This analysis will examine specific areas where time reductions or cost savings can be achieved.
- Risk Mitigation and Value Creation • Principle: AI-driven compliance models can serve as an investment in risk mitigation by reducing the likelihood of regulatory breaches and penalties. • Application: Simulation Modelling in WP3 will assess the RAG model’s adaptability to regulatory updates, offering insights into its capacity for risk reduction. These findings will provide a foundation for evaluating the model’s potential value in enhancing compliance reliability.
- Information Asymmetry Reduction • Principle: AI has the potential to reduce information asymmetry by improving access to regulatory information and providing timely, relevant guidance to compliance staff. • Application: Qualitative feedback from compliance staff in WP5 will provide insight into how the RAG model may improve decision-making by making complex regulatory information more accessible.
- Scalability and Economies of Scale • Principle: Economies of scale refer to the cost advantages that AI models can realise as they process larger volumes of data or tasks, potentially reducing marginal costs over time. • Application: WP3’s Simulation Modelling will test the model’s scalability, evaluating its performance across varied compliance tasks. Insights from this model will help determine whether economies of scale can be achieved within the specific context of Funds Axis Ltd.
- Adaptability and Dynamic Efficiency • Principle: Adaptability is critical in regulatory compliance, where evolving standards require systems to be flexible and responsive. • Application: The study will assess the RAG model’s adaptability to regulatory changes through WP3, examining the economic implications of maintaining a system that can adapt to new or updated regulations.
Summary of Theoretical Framework
The application of these economic principles will provide Funds Axis Ltd. with an evidence-based assessment of the RAG model’s feasibility, balancing cost-efficiency with practical considerations of adaptability and risk mitigation. This framework is intended to offer a well-rounded perspective, supporting informed decisions on the potential role of AI in enhancing compliance capabilities.
Project Timeline and Gantt Chart
To provide an overview of the project’s workflow, deliverables, and timeline, the following Gantt chart (Figure 1) illustrates the sequence and duration of each work package, alongside key economic analysis milestones. This layout ensures that all project phases are well-coordinated, supporting a comprehensive evaluation of the RAG model’s feasibility.
Key Aspects of the Project Timeline
- Integrated Economic Analysis Milestones: • M1 - Baseline Process Mapping & Initial CBA Report (WP1): Establishes baseline data for evaluating time and cost impacts. • M2 - Q/A CBA Assessment (WP2): Evaluates initial economic impacts of AI-driven question-answering, focusing on time and cost savings. • M3 - Simulation Model for Scalability (WP3): Tests model adaptability across different regulatory scenarios, providing insights into scalability potential. • M4 - Qualitative Interviews & Final CBA (WP5): Concludes with a synthesis of cost-benefit and user feedback data, providing a comprehensive evaluation of the RAG model’s practical and economic feasibility.
- Project Flow and Resource Allocation: • Each WP builds on the outputs of preceding phases, creating a structured sequence for cumulative insights. Economic analysis tasks are aligned with technical deliverables to provide continuous validation of project goals.
- Extended Timeline with Partner Support: • Following the initial 12-month funded period, the project enters a refinement and integration phase, including user testing and trials with Funds Axis Ltd. This phase will ensure that the final AI-enhanced module meets practical requirements and is ready for real-world deployment.
This timeline ensures that each work package and milestone aligns with the project’s objectives, facilitating a well-organised workflow that supports efficient project management. The Gantt chart provides a visual reference for all stakeholders, helping track progress and identify key points for review and adjustment.
Implementation Strategy
The implementation strategy for this project focuses on a structured, phased approach to ensure that both technical and economic assessments of the RAG model are aligned with the needs and constraints of Funds Axis Ltd. This strategy prioritises a balanced evaluation of operational feasibility and economic impact, supporting data-driven insights for decision-making.
- Embedding Economic Analysis Across Work Packages
• Objective: Integrate economic analysis within each WP to provide a comprehensive view of the RAG model’s potential impact on compliance costs, efficiency, and scalability. • Approach: Each WP will include specific economic assessment tasks, ensuring that technical progress is evaluated alongside economic insights. Interdisciplinary meetings between economists and technical experts will support this alignment. • Outcome: Continuous validation of the RAG model’s economic feasibility, providing early insights into both practical and cost-related impacts.
- Establishment of Clear Metrics and Evaluation Criteria
• Objective: Develop metrics that connect technical performance indicators (e.g., model accuracy, time savings) with economic measures (e.g., cost efficiencies, resource allocation). • Approach: Standardised metrics will be created for each WP to allow consistent data collection. Metrics will focus on key factors such as time efficiency, accuracy, usability, and cost-effectiveness. • Outcome: A structured data set to support CBA, Process Mapping, and Simulation Modelling, providing a clear comparison of current versus AI-supported processes.
- Phased Economic Modelling and Data Collection
• Objective: Support phased economic analysis by gathering data continuously as technical development progresses. • Approach: Initiate baseline CBA and Process Mapping in WP1, with ongoing data collection throughout each WP. Simulation Modelling will focus on adaptability and scalability, while final qualitative insights will be collected in WP5. • Outcome: Incremental economic evaluation to support iterative refinement and provide a holistic view of RAG’s impact at each stage of the project.
- Stakeholder Collaboration and Regular Knowledge Exchange
• Objective: Ensure that Funds Axis Ltd. is actively engaged in the project, with opportunities for feedback and knowledge exchange throughout. • Approach: Regular meetings will provide opportunities for Funds Axis Ltd. to give feedback on model usability, relevance, and alignment with current compliance needs. • Outcome: Continuous alignment of project outputs with the firm’s operational requirements, supporting user-informed development.
- User Testing and Economic Validation in Compliance Contexts
• Objective: Validate economic and operational performance through real-world testing, with a focus on both efficiency and usability. • Approach: User testing phases in WP5 will focus on practical feedback from compliance staff. Economic impacts, particularly on time savings and usability, will be a core focus during testing. • Outcome: Evidence-based insights into user experience and economic feasibility, supporting the final evaluation of the RAG model’s performance.
- Ongoing Risk Assessment and Adaptive Strategy
• Objective: Identify and mitigate technical, operational, and economic risks throughout the project, including data quality and regulatory alignment. • Approach: Establish a risk assessment framework to monitor potential challenges in each WP, with regular reviews to assess data reliability and model performance. • Outcome: Risk-informed project development, ensuring that the RAG model remains aligned with regulatory and operational standards.
- Benchmark Testing and Error Analysis
• Objective: Evaluate the RAG model’s accuracy and reliability through standardised benchmark tests and systematic error analysis. • Approach: Use financial QA benchmarks to assess model performance. Implement error analysis protocols to categorise and address common failure points. • Outcome: Enhanced model accuracy and reliability, supporting confidence in the RAG model’s potential to add value in compliance settings.
This strategy combines technical and economic assessment methods, enabling a structured evaluation of the RAG model’s operational and cost-effectiveness within the compliance framework. By embedding these considerations throughout the project phases, the strategy aims to deliver a comprehensive feasibility report that is both practical and aligned with the needs of Funds Axis Ltd.
Potential Challenges and Limitations
While the project aims to deliver a comprehensive feasibility assessment, several potential challenges and limitations may impact the implementation of the RAG model in regulatory compliance. Addressing these issues proactively will help mitigate potential risks and ensure that the findings remain practical and relevant.
• Data quality and availability: The RAG model’s effectiveness depends on the quality and completeness of the regulatory data used for training. Smaller firms may have limited access to comprehensive regulatory data across various jurisdictions, which could affect the model’s performance. To address this, the project will focus on priority compliance tasks and, where possible, supplement data with external regulatory resources.
• Model interpretability and user trust: Compliance professionals may have concerns about the interpretability of AI-driven compliance recommendations, which could impact trust and adoption. Providing clear explanations of the RAG model’s outputs will be essential to ensuring user confidence. Integrating explainable AI (XAI) techniques and providing user-friendly explanations are planned to support transparency.
• Regulatory acceptance and auditability: Ensuring that the RAG model meets regulatory requirements and can be audited is critical in compliance contexts. Regulatory bodies may require a clear audit trail for AI-driven decisions. This project will include logging and auditing features in the RAG model to record outputs and decision pathways, supporting transparency and alignment with regulatory standards.
• Economic feasibility for SMEs While RAG models have the potential to improve efficiency, initial costs for implementation and ongoing maintenance could be a barrier for smaller firms. The project’s cost-benefit analysis will provide insights into expected cost savings and long-term economic viability, offering practical recommendations for phased or modular implementation if necessary.
• Integration with existing systems: Funds Axis Ltd. may already have established compliance workflows and tools. Integrating the RAG model with these existing systems may require additional technical adaptation. The project team will conduct an early compatibility assessment and develop a modular RAG framework to enable seamless or complementary integration with current systems.
• Continuous learning and adaptation: Regulatory environments are constantly evolving, requiring compliance tools to be regularly updated to maintain accuracy. The project team will explore options for modular updates to keep the RAG model current with regulatory changes, reducing the operational burden on compliance teams while ensuring adaptability.
• Limited pool of compliance staff for qualitative data collection: Funds Axis Ltd. has a small compliance team, which may limit the sample size for qualitative insights. To address this, the project will conduct in-depth interviews and follow-up sessions with available participants to gather detailed feedback over multiple data collection points, helping to capture valuable user perspectives.
Each of these challenges represents a potential area for adjustment throughout the project. By anticipating these limitations, the project strategy includes ongoing risk assessments to ensure that both technical and economic feasibility remain central to development and final recommendations.
Ethical Implications of AI in Regulatory Compliance
The deployment of AI models, especially in regulatory compliance, raises several ethical considerations. Ensuring that the RAG model is implemented responsibly will be essential for its acceptance and effectiveness. Key ethical issues and mitigation strategies are outlined below.
• Fairness and Bias: AI models may unintentionally introduce or reinforce biases present in the training data, potentially leading to inconsistent compliance outcomes. To mitigate this, the project will carefully curate the dataset, prioritising diverse regulatory sources, and will use fairness metrics to monitor for potential bias in outputs.
• Accountability and Responsibility: Defining accountability is critical, particularly when an AI model is used to support high-stakes compliance decisions. The project will establish clear documentation to track AI-assisted decisions, ensuring that responsibility between human operators and the AI system is well-defined. This approach will support accountability and transparency.
• Privacy and Data Protection: Compliance tools often process sensitive and proprietary data, raising concerns around data privacy. The RAG model will incorporate privacy-by-design principles, including robust encryption and secure access protocols, to ensure that data protection standards, such as GDPR, are maintained throughout its use.
• Transparency and Explainability: Ensuring that the AI model’s outputs are understandable is crucial for compliance officers to trust and effectively use the RAG system. To address this, the project will implement explainable AI (XAI) techniques, providing transparent explanations of how recommendations are generated. This will support informed decision-making by compliance staff.
• Human Oversight and Job Impact: AI tools can significantly enhance efficiency, but there may be concerns about automation impacting job roles in compliance. This project emphasises that the RAG model is intended as a support tool to assist, rather than replace, compliance professionals. Human oversight will remain central to the workflow, maintaining a balance between AI-driven efficiency and the expertise of compliance staff.
• Regulatory Compliance of AI Systems: AI-driven compliance tools themselves must meet certain regulatory and ethical standards. To ensure that the RAG model is aligned with these requirements, the project will incorporate regular reviews of regulatory guidelines on AI use in financial compliance. Where guidelines are ambiguous, best practices will be followed to maintain high ethical standards.
By proactively addressing these ethical considerations, the project aims to ensure that the RAG model aligns with principles of fairness, accountability, and privacy. These measures will help build trust among compliance officers and stakeholders, supporting responsible AI use in regulatory compliance.
Future Work Ideas
Building on the insights from this feasibility study, several directions for future research and development could further enhance the capabilities of RAG models in regulatory compliance. These ideas focus on extending the applicability, reliability, and adaptability of AI models within the compliance landscape.
• Specialised Financial Language Models: Future research could explore the development of language models specifically trained on financial and regulatory texts. Such models would be better equipped to handle complex regulatory language and reasoning, potentially improving accuracy and interpretability for high-stakes compliance tasks.
• Enhanced Retrieval Mechanisms: Improving the retrieval accuracy of RAG models would help ensure that the most relevant regulatory information is surfaced in response to compliance queries. Advanced retrieval techniques, such as semantic search or context-aware filtering, could further reduce inaccuracies and improve relevance.
• Mitigating Hallucinations in High-Stakes Applications: Developing systematic methods to reduce hallucinations (incorrect or fabricated outputs) will be essential for ensuring that RAG models provide reliable guidance. Fact-checking algorithms and integration with verified knowledge bases could be effective in improving output accuracy.
• Integration of Structured Financial Data: Regulatory compliance often requires handling structured data, such as financial statements. Future work could focus on integrating structured and unstructured data within RAG models, enhancing their ability to perform tasks that involve numerical reasoning and cross-referencing.
• Longitudinal Studies on Economic Impact: Conducting long-term studies to assess the ongoing economic impact of RAG models would provide valuable insights into cost efficiencies and scalability. By tracking performance over time, future research could quantify sustained benefits, such as reduced error rates and ongoing compliance efficiencies.
• Ethical and Regulatory Guidelines for AI in Compliance: As AI adoption in compliance grows, developing ethical frameworks and best practices specific to this context will be essential. Future efforts could focus on creating industry standards for fairness, transparency, and accountability in AI-driven compliance.
These areas for future work could help to refine and expand the applicability of RAG models, creating more reliable and flexible compliance tools. These research directions would provide a foundation for ongoing advancements in AI applications for regulatory compliance, supporting a more efficient and responsive compliance landscape.
Summary and Aims for Progress
This feasibility study represents an initial step towards exploring the potential of RAG models to support and enhance regulatory compliance processes. By focusing on practical evaluation through structured work packages, the project aims to deliver actionable insights into the technical, operational, and economic feasibility of adopting AI-driven compliance tools within the context of a small FinTech firm.
The project’s structured approach, which integrates technical development with economic analysis, is designed to assess both the immediate benefits and broader implications of RAG model implementation. Through ongoing collaboration with Funds Axis Ltd., the project seeks to align its efforts with real-world compliance needs, ensuring that the evaluation reflects both the challenges and opportunities associated with this technology.
As the study progresses, particular attention will be paid to:
- Capturing baseline data to establish benchmarks for time and cost efficiencies.
- Evaluating the adaptability and scalability of RAG models in responding to evolving regulatory requirements.
- Gathering qualitative insights from compliance professionals to understand usability and practical relevance.
- Addressing potential challenges, including data quality, model interpretability, and integration with existing systems, to ensure a realistic assessment.
The findings from this study will provide a foundation for informed decision-making, offering Funds Axis Ltd. and other stakeholders a clearer understanding of how AI might enhance regulatory compliance in the financial sector. While the study does not aim to provide definitive conclusions, it aspires to create a robust framework for evaluating the potential of RAG models in this critical area of compliance.